What’s flawed isn’t educational accountability itself, but our assumption that it’s both an appropriate and adequate response for the educational challenges at hand.

What are the educational challenges at hand?

• There are obviously many.
• I focus my attention on the canonical challenges related to outputs.
  • The challenge of chronic underperformance.
  • The challenge of chronic achievement gaps.
• Since at least NCLB, the challenge of raising achievement while eliminating achievement gaps has been a pillar of education policy and accountability.
• Not surprisingly, annual state reporting of summative assessment results presumes these two challenges.
• There are obviously many educational challenges related to inputs and process that impact these outputs. Without jumping ahead too far, in general I’m skeptical of the incorporation of inputs as a solution to the current accountability conundrum – rearranging the deck chairs on the Titanic.

Two Fundamental Flaws of Current Accountability Systems

1. The presumption that educational accountability is an adequate and acceptable response to the challenge of chronic underperformance and large achievement gaps.
2. In trying to fulfill the achievement driven mandates of NCLB and ESSA, accountability systems were conceptualized and built on a flawed view of how student growth and achievement relate to each other —- making it impossible for those systems to fulfill their intended goals.

Fundamental Flaw #1

Educational accountability is an adequate and acceptable response to the challenge of chronic underperformance and large achievement gaps.

• States have spent considerable resources over the last 25 years complying with NCLB/ESSA constructing accountability systems in an attempt to fulfill this prophecy.
• From the creation of theories of action to the incorporation of growth to the current discussion of the incorporation of alternative indicators, there is a persistent belief that accountability system design is the problem to overcome.
• Looked at historically, the results of accountability systems addressing the challenge of chronic underperformance and large achievement gaps have been disappointing even to the most ardent supporters of accountability.
• Accountability, I would argue, has been and always will be inadequate to the challenge of chronic underperformance and large achievement gaps.

Fundamental Flaw #1

Educational accountability is an adequate and acceptable response to the challenge of chronic underperformance and large achievement gaps.

• Accountability systems have largely relied on public reporting and school ratings to incentivize change.
• This approach, while politically expedient, assumes that transparency and stigma alone will drive school improvement.
• Extensive research (e.g., Koretz, 2017; Bryk et al., 2015) has shown that without meaningful analytic and organizational support, high-stakes ratings often result in defensive behavior (e.g., narrowing curriculum, gaming metrics) rather than sustained capacity-building.
• The solution isn’t different or more accountability, it moving beyond accountability toward performance management.

What’s the difference between accountability and performance management?

This is the difference between a retrospective question of identifying fault as opposed to a prospective strategy to engineer some corrective measure, almost independent of considering whether there was blame-worthiness. And to move away from the blame-worthiness paradigm toward something that is more regulatory in nature. Where one might seize upon disparities or circumstances that are for some reason deemed unacceptable and engineer the interventions needed to bring about the necessary change. … It’s the no-fault gap closing strategy in which the effort is to build a consensus about a vision of an improved society rather than figure out where’s the person or entity we want to pillory.

Christopher Edley (2006)

Performance Management versus Accountability

• Performance management is a comprehensive, ongoing effort to improve an organization’s effectiveness, efficiency, and outcomes.
• Accountability is just one component of performance management—focused on measuring, reporting, and ensuring responsibility for outcomes.
• While performance management emphasizes continuous improvement, accountability primarily evaluates past performance.
• Effective performance management uses accountability as a tool, but also incorporates support, innovation, and strategic planning.
• Accountability without performance management can lead to compliance-driven behaviors rather than meaningful improvement.
• A coherent system integrates accountability within a broader performance management framework, ensuring data is used to drive improvement rather than just to judge performance.

Fundamental Flaw #2

Incoherence between the components of the accountability system and its intended goals.

• NCLB, later followed by ESSA, place central importance on the achievement of students often reported in terms of proficiency rates.
• Proficiency rates, from school accountability to the reporting of summary results at the state level assume primary significance as THE indicator of education performance.
• The primacy of proficiency rates (i.e. achievement) followed by a later addition of growth metrics into accountability systems (e.g., SGPs) has led to a fundamental misunderstanding of the relationship between achievement and growth.
• This misunderstanding manifests itself in poor performance management: Proficiency and proficiency trends are the wrong tools for the effective performance management necessary to improve an education system.

Fundamental Flaw #2

Incoherence between the components of the accountability system and its intended goals.

• The incorporation of growth metrics into accountability systems indicates that achievement and growth are viewed as different concepts.
• Yet, the difference in terms of how they are used in accountability systems belies a simple underlying relationship.
  • Growth is a leading indicator of achievement.
  • Achievement is a lagging indicator of growth.
• That is, high rates of growth (i.e., learning) lead to higher rates of attainment.
• A coherent understanding between the relationship between growth and achievement is critical to understanding performance management in education.

Coherent Components

Why Coherence is Critical?

The Reign of Proficiency

• During the last week of January, National Assessment of Educational Progress (NAEP) results were released.
• If you read beyond the headlines, an implicit message about communicating the performance of our education system is crystal clear: Proficiency reigns supreme.
• Since at least the NCLB era, education policy has been fixated on proficiency levels and their trends (i.e., achievement/attainment).
• Call it what you will —- proficiency, mastery, college and career readiness -— student attainment is the primary measure of educational performance.
• Proficiency is King! And it should be!
• Why? Claiming an education system is “good” when only a small fraction of students are proficient is absurd.
• But here’s the catch: Proficiency and proficiency trends are the wrong tools for the effective performance management necessary to improve an education system.
• Death to the King! Long Live the King!

Death to Proficiency!

• Since at least the No Child Left Behind (NCLB) era, attainment has had its detractors.
• Rightly, these critics have pointed out that using attainment for accountability purposes (e.g., school ratings) is problematic due to selection bias – students are not randomly assigned to schools.
• The solution to this problem has been to incorporate growth or value-added into the accountability system.
• Over the last 15 years, one of the primary foci of accountability system design has been weighting attainment and growth components to create more “valid” accountability indicator systems.
  
• The result, I would argue, is a design process that treats these two components like food items at a buffet.
• And today several efforts to create more valid accountability systems promote adding more components.
• The result of this process of adding more components, I contend, is incoherence in the design of accountability systems.

Attainment & Learning

• And as indicators used for accountability have increased, a significant part of the ensuing incoherence has been the imprecise, inconsistent, and incorrect use of terminology.
• Just considering results from large-scale assessments, the number of different terms is daunting:
  • Achievement, proficiency, attainment, status, and mastery are several terms frequently encountered to discuss what I refer to as attainment, a characterization of what a student or groups of students know and can do at a point in time.
  • Growth, value-added, learning, SGPs, value-tables, content-referenced growth are several terms used to characterize student growth of a student or group of students.
  • Improvement is a term that seems to be used to describe a beneficial change (either for a student or a group of students).
  • Acceleration is a term that came to prominence during the pandemic to motivate goals to correct learning loss by accelerating student learning.
• Having attended hundreds of meetings and presentations over the last decade, I can’t count the number of times I’ve heard several of these terms used in ways that have led me to believe there is a fundamental lack of understanding about how large scale assessment results should be used.

Coherence in Accountability System Design

• There is no shortage of criticism of current accountability systems.
• A substantial part of many criticisms is the over-reliance on the results from large-scale assessments.
• I would argue that the root of the problem isn’t the over-reliance on the results from large-scale assessments, but rather the incoherent way in which results from large-scale assessments are utilized.
• Accountability systems are the result of policy initiatives: A coherent accountability system ideally connects the larger policy goals of the education system (increased rates of proficiency and the closing of achievement gaps) with the processes associated with achieving (or not achieving) those goals.
• Even with coherence, there is no guarantee that the education system will achieve the policy goals.
• But without coherence, I contend, accountability initiatives intended to improve the education system are doomed to fail.

–